People's collections of electronically stored multimedia objects (also called assets) are constantly growing, and so is the need to quickly organize and search through them. Examples of multimedia objects would include digital photographs, digital video files and digital audio files. One of the most natural ways to do so is by queries in natural language. The queries can be used to search through the textual descriptions that the user has provided for the objects. Alternatively, it is possible to search through automatically generated textual description based e.g. on image and face recognition. Thus, one may retrieve photographs related to Christmas by specifying the search string “Christmas.”
This type of approach has been investigated extensively. For example, it is described for in U.S. Pat. No. 5,493,677 by Balogh et al., entitled “Generation, Archiving and Retrieval of Digital Images with Evoked Suggestion-Set Captions and Natural Language Interface” and U.S. Pat. No. 6,233,547 by M. Debner, entitled “Computer Program Product for Retrieving Multi-Media Objects Using a Natural Language Having a Pronoun”. It is also used in most state-of-the-art internet search engines, such as google (www.google.com) and bing (www.bing.com).
The simplest of such approaches (e.g. google's search engine) look for matches between words from the query and the words in each description, and augment the technique by taking into account a list of synonyms (e.g. currently, in google, the query “U.S.A.” also matches “University of St. Augustine”). The most advanced approaches apply natural language techniques in order to understand the meaning of both query and description, and then use some form of inference to see if their meanings match. So, for example they can tell that there is a match between the query “Frank on a plane” and the description “Frank in the cockpit”, while there is no match between “Frank on a plane” and “Frank stepping on his paper plane”.
To complicate matters, the interpretation of the descriptions provided for multimedia assets often requires external knowledge. This is particularly important when knowledge about social networks is involved. For example, consider a social network in which John is Sara's boss, and is also Cindy's father. The query “Sara's supervisor” obviously matches the asset description “Cindy with her dad”, but only if the knowledge from the social network is taken into account. This situation is particularly common in multimedia collections about families, where, for example, “Frank's father” may also be “Jim's grandfather”.
None of the search techniques mentioned above takes into account social network information in the matching process.
Consequently, a need exists for a retrieval system that takes into account social network information in the matching process.